--- title: IceVision Bboxes - Real Data keywords: fastai sidebar: home_sidebar nb_path: "nbs/iv_bbox_real.ipynb" ---
{% raw %}
{% endraw %}

This is a mashup of IceVision's "Custom Parser" example and their "Getting Started (Object Detection)" notebooks, to analyze SPNet Real dataset, for which I generated bounding boxes. -- S.H. Hawley, July 1, 2021

Installing IceVision and IceData

If on Colab run the following cell, else check the installation instructions

{% raw %} 
#try:
#    !wget https://raw.githubusercontent.com/airctic/icevision/master/install_colab.sh
#    !chmod +x install_colab.sh && ./install_colab.sh
#except:
#    print("Ignore the error messages and just keep going")
{% endraw %} {% raw %}
{% endraw %} {% raw %} 
import torch, re 
tv, cv = torch.__version__, torch.version.cuda
tv = re.sub('\+cu.*','',tv)
TORCH_VERSION = 'torch'+tv[0:-1]+'0'
CUDA_VERSION = 'cu'+cv.replace('.','')

print(f"TORCH_VERSION={TORCH_VERSION}; CUDA_VERSION={CUDA_VERSION}")
print(f"CUDA available = {torch.cuda.is_available()}, Device count = {torch.cuda.device_count()}, Current device = {torch.cuda.current_device()}")
print(f"Device name = {torch.cuda.get_device_name()}")
print("hostname:")
!hostname

TORCH_VERSION=torch1.8.0; CUDA_VERSION=cu102
CUDA available = True, Device count = 1, Current device = 0
Device name = TITAN X (Pascal)
hostname:
lecun
{% endraw %} {% raw %} 
#!pip install -qq mmcv-full=="1.3.8" -f https://download.openmmlab.com/mmcv/dist/{CUDA_VERSION}/{TORCH_VERSION}/index.html --upgrade
#!pip install mmdet -qq
{% endraw %}

Imports

As always, let's import everything from icevision. Additionally, we will also need pandas (you might need to install it with pip install pandas).

{% raw %} 
from icevision.all import *
import pandas as pd

INFO     - The mmdet config folder already exists. No need to downloaded it. Path : /home/shawley/.icevision/mmdetection_configs/mmdetection_configs-2.10.0/configs | icevision.models.mmdet.download_configs:download_mmdet_configs:17
{% endraw %}

Download dataset

We're going to be using a small sample of the chess dataset, the full dataset is offered by roboflow here

{% raw %} 
#data_dir = icedata.load_data(data_url, 'chess_sample') / 'chess_sample-master'

# SPNET Real Dataset link (currently proprietary, thus link may not work)
#data_url = "https://hedges.belmont.edu/~shawley/spnet_sample-master.zip"
#data_dir = icedata.load_data(data_url, 'spnet_sample') / 'spnet_sample-master' 

# public espiownage cyclegan dataset:
#data_url = 'https://hedges.belmont.edu/~shawley/espiownage-cyclegan.tgz'
#data_dir = icedata.load_data(data_url, 'espiownage-cyclegan') / 'espiownage-cyclegan'

# local data already there:
from pathlib import Path
data_dir = Path('/home/shawley/datasets/espiownage-cleaner')  # real data is local and private
{% endraw %}

Understand the data format

In this task we were given a .csv file with annotations, let's take a look at that.

!!! danger "Important"
Replace source with your own path for the dataset directory.

{% raw %} 
df = pd.read_csv(data_dir / "bboxes/annotations.csv")
df.head()
filename width height label xmin ymin xmax ymax
0 06240907_proc_00254.png 512 384 1 31 135 184 290
1 06240907_proc_00256.png 512 384 0 65 153 168 270
2 06240907_proc_00270.png 512 384 1 45 149 164 280
3 06240907_proc_00281.png 512 384 10 0 111 185 340
4 06240907_proc_00281.png 512 384 1 254 134 353 215
{% endraw %}

At first glance, we can make the following assumptions:

  • Multiple rows with the same filename, width, height
  • A label for each row
  • A bbox [xmin, ymin, xmax, ymax] for each row

Once we know what our data provides we can create our custom Parser.

{% raw %} 
set(np.array(df['label']).flatten())

{0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11}
{% endraw %} {% raw %} 
#df['label'] = ["Object"]*len(df)#  "_"+df['label'].apply(str)   # force label to be string-like
{% endraw %} {% raw %} 
df['label'] /= 2
#df.head()
df['label'] = df['label'].apply(int) 
print(set(np.array(df['label']).flatten()))
df['label'] = "_"+df['label'].apply(str)+"_"

{0, 1, 2, 3, 4, 5}
{% endraw %} {% raw %} 
df.head()
filename width height label xmin ymin xmax ymax
0 06240907_proc_00254.png 512 384 _0_ 31 135 184 290
1 06240907_proc_00256.png 512 384 _0_ 65 153 168 270
2 06240907_proc_00270.png 512 384 _0_ 45 149 164 280
3 06240907_proc_00281.png 512 384 _5_ 0 111 185 340
4 06240907_proc_00281.png 512 384 _0_ 254 134 353 215
{% endraw %} {% raw %} 
df['label'] = 'AN'  # antinode
df.head()
filename width height label xmin ymin xmax ymax
0 06240907_proc_00254.png 512 384 AN 31 135 184 290
1 06240907_proc_00256.png 512 384 AN 65 153 168 270
2 06240907_proc_00270.png 512 384 AN 45 149 164 280
3 06240907_proc_00281.png 512 384 AN 0 111 185 340
4 06240907_proc_00281.png 512 384 AN 254 134 353 215
{% endraw %}

Create the Parser

The first step is to create a template record for our specific type of dataset, in this case we're doing standard object detection:

{% raw %} 
template_record = ObjectDetectionRecord()
{% endraw %}

Now use the method generate_template that will print out all the necessary steps we have to implement.

{% raw %} 
Parser.generate_template(template_record)

class MyParser(Parser):
    def __init__(self, template_record):
        super().__init__(template_record=template_record)
    def __iter__(self) -> Any:
    def __len__(self) -> int:
    def record_id(self, o: Any) -> Hashable:
    def parse_fields(self, o: Any, record: BaseRecord, is_new: bool):
        record.set_img_size(<ImgSize>)
        record.set_filepath(<Union[str, Path]>)
        record.detection.add_bboxes(<Sequence[BBox]>)
        record.detection.set_class_map(<ClassMap>)
        record.detection.add_labels(<Sequence[Hashable]>)
{% endraw %}

We can copy the template and use it as our starting point. Let's go over each of the methods we have to define:

  • __init__: What happens here is completely up to you, normally we have to pass some reference to our data, data_dir in our case.

  • __iter__: This tells our parser how to iterate over our data, each item returned here will be passed to parse_fields as o. In our case we call df.itertuples to iterate over all df rows.

  • __len__: How many items will be iterating over.

  • imageid: Should return a Hashable (int, str, etc). In our case we want all the dataset items that have the same filename to be unified in the same record.

  • parse_fields: Here is where the attributes of the record are collected, the template will suggest what methods we need to call on the record and what parameters it expects. The parameter o it receives is the item returned by __iter__.

!!! danger "Important"
Be sure to pass the correct type on all record methods!

{% raw %} 
class BBoxParser(Parser):
    def __init__(self, template_record, data_dir):
        super().__init__(template_record=template_record)
        
        self.data_dir = data_dir
        self.df = pd.read_csv(data_dir / "bboxes/annotations.csv")
        #self.df['label'] /= 2
        #self.df['label'] = self.df['label'].apply(int) 
        #self.df['label'] = "_"+self.df['label'].apply(str)+"_"
        self.df['label'] = 'AN'  # make them all the same object
        self.class_map = ClassMap(list(self.df['label'].unique()))
        
    def __iter__(self) -> Any:
        for o in self.df.itertuples():
            yield o
        
    def __len__(self) -> int:
        return len(self.df)
        
    def record_id(self, o) -> Hashable:
        return o.filename
        
    def parse_fields(self, o, record, is_new):
        if is_new:
            record.set_filepath(self.data_dir / 'images' / o.filename)
            record.set_img_size(ImgSize(width=o.width, height=o.height))
            record.detection.set_class_map(self.class_map)
        
        record.detection.add_bboxes([BBox.from_xyxy(o.xmin, o.ymin, o.xmax, o.ymax)])
        record.detection.add_labels([o.label])
{% endraw %}

Let's randomly split the data and parser with Parser.parse:

{% raw %} 
parser = BBoxParser(template_record, data_dir)
{% endraw %} {% raw %} 
train_records, valid_records = parser.parse()

INFO     - Autofixing records | icevision.parsers.parser:parse:136
{% endraw %}

Let's take a look at one record:

{% raw %} 
show_record(train_records[5], display_label=False, figsize=(14, 10))
{% endraw %} {% raw %} 
train_records[0]

BaseRecord

common: 
	- Record ID: 58
	- Image size ImgSize(width=512, height=384)
	- Filepath: /home/shawley/datasets/espiownage-cleaner/images/06240907_proc_00417.png
	- Img: None
detection: 
	- BBoxes: [<BBox (xmin:0, ymin:105, xmax:192, ymax:338)>, <BBox (xmin:229, ymin:161, xmax:352, ymax:262)>]
	- Class Map: <ClassMap: {'background': 0, 'AN': 1}>
	- Labels: [1, 1]
{% endraw %}

Moving On...

Following the Getting Started "refrigerator" notebook...

{% raw %} 
# size is set to 384 because EfficientDet requires its inputs to be divisible by 128
image_size = 384  
train_tfms = tfms.A.Adapter([*tfms.A.aug_tfms(size=image_size, presize=512), tfms.A.Normalize()])
valid_tfms = tfms.A.Adapter([*tfms.A.resize_and_pad(image_size), tfms.A.Normalize()])

# Datasets
train_ds = Dataset(train_records, train_tfms)
valid_ds = Dataset(valid_records, valid_tfms)
{% endraw %} {% raw %} 
samples = [train_ds[0] for _ in range(3)]
show_samples(samples, ncols=3)
{% endraw %} {% raw %} 
model_type = models.mmdet.retinanet
backbone = model_type.backbones.resnet50_fpn_1x(pretrained=True)
{% endraw %} {% raw %} 
selection = 0

extra_args = {}

if selection == 0:
  model_type = models.mmdet.retinanet
  backbone = model_type.backbones.resnet50_fpn_1x

elif selection == 1:
  # The Retinanet model is also implemented in the torchvision library
  model_type = models.torchvision.retinanet
  backbone = model_type.backbones.resnet50_fpn

elif selection == 2:
  model_type = models.ross.efficientdet
  backbone = model_type.backbones.tf_lite0
  # The efficientdet model requires an img_size parameter
  extra_args['img_size'] = image_size

elif selection == 3:
  model_type = models.ultralytics.yolov5
  backbone = model_type.backbones.small
  # The yolov5 model requires an img_size parameter
  extra_args['img_size'] = image_size

model_type, backbone, extra_args

(<module 'icevision.models.mmdet.models.retinanet' from '/home/shawley/envs/icevision/lib/python3.8/site-packages/icevision/models/mmdet/models/retinanet/__init__.py'>,
 <icevision.models.mmdet.models.retinanet.backbones.resnet_fpn.MMDetRetinanetBackboneConfig at 0x7f6ac9451700>,
 {})
{% endraw %} {% raw %} 
model = model_type.model(backbone=backbone(pretrained=True), num_classes=len(parser.class_map), **extra_args) 

/home/shawley/envs/icevision/lib/python3.8/site-packages/mmdet/core/anchor/builder.py:16: UserWarning: ``build_anchor_generator`` would be deprecated soon, please use ``build_prior_generator`` 
  warnings.warn(

Use load_from_local loader
The model and loaded state dict do not match exactly

size mismatch for bbox_head.retina_cls.weight: copying a param with shape torch.Size([720, 256, 3, 3]) from checkpoint, the shape in current model is torch.Size([9, 256, 3, 3]).
size mismatch for bbox_head.retina_cls.bias: copying a param with shape torch.Size([720]) from checkpoint, the shape in current model is torch.Size([9]).
{% endraw %} {% raw %} 
train_dl = model_type.train_dl(train_ds, batch_size=8, num_workers=4, shuffle=True)
valid_dl = model_type.valid_dl(valid_ds, batch_size=8, num_workers=4, shuffle=False)
{% endraw %} {% raw %} 
model_type.show_batch(first(valid_dl), ncols=4)
{% endraw %} {% raw %} 
metrics = [COCOMetric(metric_type=COCOMetricType.bbox)]
{% endraw %} {% raw %} 
learn = model_type.fastai.learner(dls=[train_dl, valid_dl], model=model, metrics=metrics)
{% endraw %} {% raw %} 
learn.lr_find(end_lr=0.005)

# For Sparse-RCNN, use lower `end_lr`
# learn.lr_find(end_lr=0.005)

/home/shawley/envs/icevision/lib/python3.8/site-packages/mmdet/core/anchor/anchor_generator.py:324: UserWarning: ``grid_anchors`` would be deprecated soon. Please use ``grid_priors`` 
  warnings.warn('``grid_anchors`` would be deprecated soon. '
/home/shawley/envs/icevision/lib/python3.8/site-packages/mmdet/core/anchor/anchor_generator.py:360: UserWarning: ``single_level_grid_anchors`` would be deprecated soon. Please use ``single_level_grid_priors`` 
  warnings.warn(

SuggestedLRs(lr_min=7.945936522446573e-05, lr_steep=6.597539322683588e-05)
{% endraw %} {% raw %} 
learn.fine_tune(60, 7e-5, freeze_epochs=2)
epoch train_loss valid_loss COCOMetric time
0 0.720761 0.500148 0.475057 00:51
1 0.467032 0.424952 0.526241 00:48
epoch train_loss valid_loss COCOMetric time
0 0.401495 0.366908 0.598216 00:55
1 0.387994 0.359718 0.585890 00:54
2 0.362751 0.344039 0.599286 00:53
3 0.361480 0.335245 0.616614 00:54
4 0.358037 0.337374 0.602294 00:53
5 0.348905 0.322019 0.622001 00:54
6 0.342799 0.310715 0.638099 00:53
7 0.333430 0.306771 0.638176 00:54
8 0.322321 0.304977 0.642818 00:53
9 0.319668 0.304358 0.639192 00:54
10 0.322624 0.304828 0.628034 00:53
11 0.316567 0.294524 0.643822 00:53
12 0.313344 0.305355 0.627094 00:54
13 0.307588 0.296331 0.646967 00:53
14 0.313001 0.295556 0.635726 00:53
15 0.306109 0.286957 0.650334 00:53
16 0.301551 0.290895 0.644181 00:53
17 0.298018 0.288082 0.641732 00:54
18 0.291314 0.282551 0.658020 00:53
19 0.288914 0.278754 0.656703 00:53
20 0.289636 0.282614 0.650285 00:54
21 0.285580 0.289000 0.656285 00:53
22 0.282277 0.307557 0.647870 00:54
23 0.283635 0.280513 0.661381 00:53
24 0.291228 0.274230 0.659838 00:54
25 0.278158 0.278775 0.645381 00:53
26 0.273699 0.282900 0.648760 00:53
27 0.272332 0.281317 0.651336 00:53
28 0.265779 0.275999 0.655821 00:53
29 0.271596 0.279412 0.652544 00:53
30 0.264325 0.273436 0.662996 00:53
31 0.261695 0.279709 0.654904 00:53
32 0.263747 0.275295 0.658562 00:53
33 0.258810 0.275320 0.652618 00:53
34 0.261362 0.274700 0.657423 00:53
35 0.256187 0.277283 0.658754 00:54
36 0.252854 0.276791 0.649310 00:53
37 0.254081 0.272175 0.658432 00:53
38 0.254751 0.278115 0.650505 00:53
39 0.242098 0.276294 0.652749 00:53
40 0.248349 0.274449 0.655343 00:53
41 0.245198 0.278592 0.658989 00:53
42 0.242970 0.275323 0.659475 00:53
43 0.236866 0.281375 0.649309 00:53
44 0.238895 0.274509 0.662528 00:54
45 0.236849 0.275679 0.656944 00:53
46 0.232527 0.281724 0.648647 00:54
47 0.231769 0.280163 0.652584 00:53
48 0.232933 0.280051 0.651751 00:53
49 0.232995 0.279019 0.653427 00:53
50 0.229541 0.278091 0.653912 00:53
51 0.234844 0.278182 0.654136 00:53
52 0.226265 0.280948 0.653570 00:53
53 0.240393 0.279526 0.655095 00:53
54 0.238497 0.278854 0.652016 00:53
55 0.231896 0.277662 0.654968 00:53
56 0.225483 0.278757 0.653075 00:53
57 0.233954 0.278795 0.654114 00:53
58 0.227197 0.278647 0.652862 00:53
59 0.222711 0.278647 0.653004 00:53
{% endraw %} {% raw %} 
model_type.show_results(model, valid_ds, detection_threshold=.5)
{% endraw %} {% raw %} 
learn.save('iv_bbox_real')

Path('models/iv_bbox_real.pth')
{% endraw %}

Inference

Might get a CUDA OOM error here. If so, restart kernel and load what we just saved. You'll have to go back and re-define learn, model, valid_ds etc., though.

{% raw %} 
learn.load('iv_bbox_real')

<fastai.learner.Learner at 0x7f6af37e0be0>
{% endraw %} {% raw %} 
preds = model_type.predict(model, valid_ds, keep_images=True)

/home/shawley/envs/icevision/lib/python3.8/site-packages/mmdet/core/anchor/anchor_generator.py:324: UserWarning: ``grid_anchors`` would be deprecated soon. Please use ``grid_priors`` 
  warnings.warn('``grid_anchors`` would be deprecated soon. '
/home/shawley/envs/icevision/lib/python3.8/site-packages/mmdet/core/anchor/anchor_generator.py:360: UserWarning: ``single_level_grid_anchors`` would be deprecated soon. Please use ``single_level_grid_priors`` 
  warnings.warn(
{% endraw %} {% raw %} 
show_preds(preds=preds[0:10])
{% endraw %} {% raw %} 
len(train_ds), len(valid_ds), len(preds)

(1564, 391, 391)
{% endraw %}

let's try to figure out how to get what we want from these predictions. hmmm

{% raw %} 
preds[0].pred

BaseRecord

common: 
	- Image size ImgSize(width=384, height=384)
	- Record ID: 1283
	- Img: 384x384x3 <np.ndarray> Image
detection: 
	- Scores: []
	- Class Map: <ClassMap: {'background': 0, 'AN': 1}>
	- Labels: []
	- BBoxes: []
{% endraw %} {% raw %} 
preds[1].pred.detection.scores

array([0.74782693], dtype=float32)
{% endraw %} {% raw %} 
preds[1].pred.detection.bboxes

[<BBox (xmin:157.4410400390625, ymin:131.0904541015625, xmax:273.8079528808594, ymax:248.00094604492188)>]
{% endraw %} {% raw %} 
preds[1].pred.detection.bboxes[0].xmin

def get_bblist(pred):
    my_bblist = []
    bblist = pred.pred.detection.bboxes
    for i in range(len(bblist)):
        my_bblist.append([bblist[i].xmin, bblist[i].ymin, bblist[i].xmax, bblist[i].ymax])
    return my_bblist

get_bblist(preds[1])      

[[157.44104, 131.09045, 273.80795, 248.00095]]
{% endraw %} {% raw %} 
preds[3].pred

BaseRecord

common: 
	- Image size ImgSize(width=384, height=384)
	- Record ID: 1385
	- Img: 384x384x3 <np.ndarray> Image
detection: 
	- Class Map: <ClassMap: {'background': 0, 'AN': 1}>
	- Labels: []
	- Scores: []
	- BBoxes: []
{% endraw %} {% raw %} 
results = []
for i in range(len(preds)):
    if (len(preds[i].pred.detection.scores) == 0): continue   # sometimes you get a zero box/prediction. ??
    #print(f"i = {i}, file = {str(Path(valid_ds[i].common.filepath).stem)+'.csv'}, bboxes = {get_bblist(preds[i])}, scores={preds[i].pred.detection.scores}\n")
    worst_score = np.min(np.array(preds[i].pred.detection.scores))
    line_list = [str(Path(valid_ds[i].common.filepath).stem)+'.csv', get_bblist(preds[i]), preds[i].pred.detection.scores, worst_score, i]
    results.append(line_list)
    
# store as pandas dataframe
res_df = pd.DataFrame(results, columns=['filename', 'bblist','scores','worst_score','i'])
res_df = res_df.sort_values('worst_score')  # order by worst score as a "top losses" kind of thing
res_df.head() # take a look
filename bblist scores worst_score i
16 06241902_proc_00857.csv [[101.7664, 97.137886, 214.55583, 225.27325]] [0.50051755] 0.500518 33
99 06240907_proc_00556.csv [[0.0, 34.046432, 157.73097, 326.12262]] [0.50141394] 0.501414 254
52 06240907_proc_00724.csv [[173.59372, 135.70735, 271.25345, 242.49944]] [0.502879] 0.502879 147
45 06240907_proc_00725.csv [[209.86786, 52.368305, 265.12656, 128.56065]] [0.5033824] 0.503382 126
103 06240907_proc_00299.csv [[0.0, 83.26705, 149.51791, 328.78644]] [0.5035457] 0.503546 264
{% endraw %} {% raw %} 
res_df.to_csv('bboxes_top_losses_real.csv', index=False)
{% endraw %}